Impact of observer experience on multi-detector computed tomography aortic valve morphology assessment and valve size selection for transcatheter aortic valve replacement

Transcatheter aortic valve replacement (TAVR) has become the standard treatment for aortic stenosis in older patients. It increasingly relies on accurate pre-procedural planning using multidetector computed tomography (MDCT). Since little is known about the required competence levels for MDCT analyses, we comprehensively assessed MDCT TAVR planning reproducibility and accuracy with regard to valve selection in various healthcare workers. 20 randomly selected MDCT of TAVR patients were analyzed using dedicated software by healthcare professionals with varying backgrounds and experience (two structural interventionalists, one imaging specialist, one cardiac surgeon, one general physician, and one medical student). Following the analysis, the most appropriate Edwards SAPIEN 3™ and Medtronic CoreValve valve size was selected. Intra- and inter-observer variability were assessed. The first structural interventionalist was considered as reference standard for inter-observer comparison. Excellent intra- and inter-observer variability was found for the entire group in regard to the MDCT measurements. The best intra-observer agreement and reproducibility were found for the structural interventionalist, while the medical student had the lowest reproducibility. The highest inter-observer agreement was between both structural interventionalists, followed by the imaging specialist. As to valve size selection, the structural interventionalist showed the highest intra-observer reproducibility, independent of the brand of valve used. Compared to the reference structural interventionalist, the second structural interventionalist showed the highest inter-observer agreement for valve size selection [ICC 0.984, 95% CI 0.969–0.991] followed by the cardiac surgeon [ICC 0.947, 95%CI 0.900–0.972]. The lowest inter-observer agreement was found for the medical student [ICC 0.507, 95%CI 0.067–0.739]. While current state-of-the-art MDCT analysis software provides excellent reproducibility for anatomical measurements, the highest levels of confidence in terms of valve size selection were achieved by the performing interventional physicians. This was most likely attributable to observer experience.

the incidence of complications such as valve prosthesis dislocation or annulus rupture by an under-or oversized prosthesis 7,8 . Due to its high spatial resolution, multidetector computed tomography (MDCT) has become the standard pre-procedural imaging method in patients undergoing TAVR 9 . Previous studies showed high levels of inter-observer agreement in terms of MDCT measurements and valve size selection, but were primarily focussing on readers with existing imaging experience without experience in structural intervention 10,11 . To date, national guidelines do not define how much imaging experience is required to analyse pre-procedural MDCT scans, or the qualifications necessary to perform the analysis in order to select an appropriate prosthesis size 12 . Therefore, we carried out a comprehensive assessment of MDCT post-processing analyses of reproducibility and accuracy of valve selection in different healthcare professionals including two structural interventionalists, an imaging specialist, a cardiac surgeon, a general physician and a medical student in a tertiary cardiology hospital in order to define minimum post-processing competence levels and provide guidance on required performance for preprocedural planning.

Methods
Twenty patients with severe AS as confirmed by echocardiography and who underwent TAVR in 2019 were enrolled; patients with bicuspid AS were excluded. This study was conducted according to the principles of the Helsinki Declaration.

Multidetector computed tomography. A dual-source CT scanner (SOMATOM Force, Siemens
Healthcare GmbH, Erlangen, Germany) was used to generate contrast-enhanced MDCT scans in a prospectively ECG-triggered high-pitch spiral acquisition mode. The region of interest extended from the clavicles to the femoral heads. CT angiography was performed with bolus tracking in the descending aorta using a contrast agent bolus of 80 ml (Imeron 350, Bracco Imaging, Konstanz, Germany) followed by a 40 ml saline chaser, both at a flow rate of 4 ml/sec. Scan parameters were as follows: 2 × 192 × 0.6 mm collimation, 250 ms rotation time, pitch of 3.2, automated tube current adaption. A small field of view data set with medium soft convolution kernel (Siemens Bv36), 0.75 mm slice thickness and 0.5 mm slice increment was generated for the assessment of the aortic annulus, root, and valve morphology and dimensions. All data were analysed using dedicated software (3 Mensio, Structural Heart, V9.1., Pie Medical Imaging, Maastricht, Netherlands).

Study design.
Five different healthcare professionals of varying levels of experience were involved in this study to assess the impact of experience on post-processing of MDCT scans. All observers focused their analysis on the structures which determine valve selection, whereas the vascular approach was not considered. The parameters obtained in this study are displayed in Table 1. The group of observers included two interventional structural heart cardiologists (reference structural interventionalist and a second one), both with more than 5 years of experience in TAVR, one cardiac imaging specialist (cardiologist by training), one cardiac surgeon with more than 5 years of experience in TAVR and SAVR, one general physician and one medical student. All observers were given a brief training on the software provided by the company. This training was comprised of a presentation on how to use the software including an exemplary demonstration of an evaluation in one patient. In addition, all observers underwent a short briefing to standardize the measurements according to the study protocol. All scans were analysed twice in a standardized approach by each observer (with exception of the structural interventionalist two), with at least 4 weeks between the two runs, to assess intra-observer variability in regard to the measured parameters (aortic annulus area, aortic annulus area derived diameter, aortic annulus diameter average, aortic annulus perimeter, sinus of valsalva diameter). The measurements of the structural interventionalists two were used to assess the inter-observer variability in terms of MDCT measurements as well as valve size selection.
After each run, each observer had to choose the valve size for the Medtronic CoreValve (23 mm, 26 mm, 29 mm, 34 mm) and the Edwards SAPIEN 3™ valve (23 mm, 26 mm, 29 mm) that were used at the Heart Centre Göttingen, at that time. The selection was based on the measurements and a recommendation sheet provided by the manufacturer (Tables 2 and 3). To determine the inter-observer variability, the first run measurements of the observers were compared to the structural interventionalist, who was considered as reference.
Statistics. Statistical  For between-group comparisons in normally distributed data, t-or ANOVA testing were carried out as appropriate. P-values provided are two-sided, an alpha level of ≤ 0.05 was considered statistically significant. Furthermore, intra-and inter-observer variability was assessed using three different methods: intraclass correlation coefficients (ICC), Bland Altman analysis, and coefficients of variation (CoV). Bland Altmann analysis reveals "mean differences". When the compared measurements revealed exactly the same result, all the differences would be equal to zero. A deviation to zero represents the average deviation of measurement x to measurement y 13 . The CoV was defined as the standard deviation of the differences divided by the mean 14,15 . The level of agreement was defined as follows: excellent for ICC > 0.74, good for ICC 0.60-0.74, for ICC 0.40-0.59, and poor for ICC < 0.4 16 . Ethical approval. This study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the local ethics committee of the University Medical Center Göttingen (10/5/16). Informed consent was obtained from all subjects involved in the study.

Results
Demographics. Patients´ characteristics are displayed in Table 4. A total of 14 (70%) patients were male.
Mean age was 78 ± 6 years and ranged from 61 to 90 years. AS was confirmed in all cases by echocardiography with a mean transvalvular peak velocity (V max) of 4.1 ± 0.7 m/s and an average transvalvular mean gradient of 40.3 ± 14.4 mmHg. The estimated mean aortic valve area was 0.7 ± 0.3 cm 2 .
Assessment of aortic anatomy for valve size selection. Mean annulus area measured by the different observers ranged between 502.87 mm 2 and 571.32 mm 2 , which resulted in significant differences between the observers (p < 0.001). The mean aortic annulus area derived diameter was measured between 25.1 ± 2.9 mm and 26.8 ± 3.3 mm and differed significantly between the observers (p < 0.001). The aortic annulus area derived annulus diameter differed numerically only minimally from the measured averaged aortic annulus diameter and ranged between 25.4 ± 2.7 mm and 27.0 ± 3.4 mm. Mean annulus perimeter varied from 80.3 ± 9.1 mm to 91.0 ± 11.9 mm with significant differences between the observers (p < 0.001). The average measured diameter of the sinus of valsalva (SOV) ranged between 33.3 ± 4.5 mm and 33.6 ± 4.9 mm (p = 0.504). Figure 1A-D illustrates the measured values by the 5 observers and also indicates significant inter-observer differences for all aforementioned parameters (online data supplement Figure S1 includes also structural interventionalist two).
Valve size selection. Overall, for the Edwards SAPIEN 3™ valve the medium-sized valve (26 mm) was chosen most frequently (49%) followed by the largest one (29 mm, 38%) and the smallest one (23 mm, 13%). was not selected by any of the operators. There was no difference on an individual observer level. Details are given in the online data supplement (online data supplement Table S1-S2).

Intra-and inter-observer variability of CT measurements. Excellent intra-observer agreement
was seen for all observers (Table 5 and online data supplement Figure S2-S6). The structural interventionalist showed the best intra-observer agreement and reproducibility for all measured MDCT parameters, with the exception of SOV. The medical student had the lowest reproducibility in 3 out of 5 categories (aortic annulus area derived diameter, aortic annulus average diameter and aortic annulus perimeter). Numerically small but statistically significant differences were observed between the initial and the repeated analysis runs for the following parameters: 1. Annulus area, aortic annulus area derived diameter, aortic annulus average diameter and aortic annulus perimeter in the measurements by the cardiac surgeon; 2. The aortic annulus perimeter in the measurements of the medical student; 3. The SOV measurements of the general physician (online data supplement Figures S7-S10).
There were no statistically significant differences between the two runs for the structural interventionalist and the imaging specialist.
Compared to the structural interventionalist the inter-observer agreement and reproducibility were excellent for all analysed data including the analyses of the structural interventionalist two, the imaging specialist, the cardiac surgeon and the general physician. In addition, the medical student showed also excellent reproducibility for 4 out of 5 recorded parameters. Only the aortic annulus perimeter agreement did not reach excellent interobserver reproducibility and was considered "good" compared to the structural interventionalist. The highest agreement with the structural interventionalist was observed for the second structural interventionalist followed by the imaging specialist, the general physician and the cardiac surgeon (Table 6).
Intra-and inter-observer variability for valve size selection. All observers reached an excellent level of intra-observer agreement for valve size selection. When estimating the ICC without taking into account the different valve manufacturers, excellent intra-observer agreements were found for all observers, with the highest intra-observer agreement for the structural interventionalist [ Table 7.  Table 7. Furthermore, Fig. 2 depicts the valve size selection agreement of the different observers and the structural interventionalist, whereas Figure S11 of the online data supplement illustrates the annulus area measurements in three selected patients where the measurements resulted in different valve size selections as compared to the structural interventionalist.

Discussion
To our knowledge, this is the first study investigating the impact of the experience of different healthcare professionals involved in the field of MDCT measurements and valve size selection prior to TAVR. We included several different healthcare professionals: two structural interventionalists, a cardiac surgeon, an imaging specialist, a general physician and a medical student. The following findings are noteworthy: First, MDCT TAVR planning analyses were feasible and robust, with minimal differences between all observers studied, after they all received a brief introduction to the analysis software. Second, among the different observers (excluding the second structural interventionalist) the imaging specialist had the highest and the medical student the lowest agreement with the reference structural interventionalist. Third, in terms of valve size selection the structural interventionalist had the highest level of consistency between repeated analyses which was superior to all other observers. This suggests that besides an adequate analysis of pre-TAVR MDCT scans, clinical and interventional experience should be a prerequisite for consistent and safe procedural planning. Taken together, high quality  www.nature.com/scientificreports/ measurements of the relevant aortic anatomy can be obtained relatively easily based on state-of-the-art MDCT TAVR planning scans, which can then be utilized for accurate determination of the intervention. It is known that with increasing experience-defined by the number of TAVR procedures performed-the rate of periprocedural complications (mortality, vascular complications, bleeding) decreases 17 . An accurate selection of the valve size also contributes to a reduced complication rate. Therefore, accurate pre-procedural imaging is crucial to assure optimal patient outcome. Transoesophageal echocardiography was initially used to estimate the right valve size, but nowadays MDCT has become the standard procedure, which has led to a reduction in paravalvular regurgitation 18,19 . In line with previous MDCT studies, our study demonstrates excellent intra-observer and inter-observer results for all observers involved, independent of the level of experience, when following a standardized analysis procedure 10,11,[19][20][21][22][23] . Due to the high measurement accuracy achieved with MDCT, it initially seems counterintuitive that there were differences in the selection of the valve sizes between and within the observers in the current study. One possible explanation may be that the structural interventionalists traced the border of the region of interest consistently within the outer contrast region, while the other observers delineated the borders around the contrast region resulting in slightly smaller measurements of the interventionalists (please see Figure S11). In cases with a high AVC load the interventionalists bisected the calcifications considering the blooming artefact, which was not as consistently performed by the other observers. Both may have impacted the measurements slightly and consequently valve size selection. A further possible explanation is, that valve size selection is not based on one single parameter but rather, when using the recommendation sheet, on up to four different parameters. There is a slight overlap where one or the other valve size may be chosen (Tables 2 and 3). Therefore, differences may be explained by cases in which the different parameters do not fit one size only, or in the presence of measurements between 2 valve sizes. Horehledova et al. showed, that only 30 to 60% of annulus diameter, annulus area and annulus perimeter measurements indicated the same prosthesis size 24 . In such cases, the level of clinical experience observer becomes more relevant. This may be why the best intra-observer agreement regarding valve size selection was observed for the structural interventionalist, who encounters similar scenarios in their clinical routine and thus targets them in a more structured manner. This may also be the explanation why the structural interventionalist always selected the same valve size in cases of borderline levels of the sizing parameters, while the other observers were not as consistent in this aspect. www.nature.com/scientificreports/ In addition, the level of experience of the observer most likely also plays a role in the measurements: a higher agreement was again seen in our study for the structural interventionalist compared to the other observers. While non-interventionalists probably relied mostly on their measured parameters and selected the valve size accordingly, the structural interventionalists may have also intuitively based their decisions on appearance and anatomy, e.g. calcium distribution. However, to reduce peri-interventional complications during TAVR, aspects additional to the dimensional measurements (Table 1) have to be considered. These include the shape and size of the aortic annulus, the LVOT ascending aorta angle as well as the extent and distribution of aortic valve calcification. A comprehensive assessment of all parameters involved make it possible to reduce the risk of paravalvular regurgitation, which is associated with worse outcome 25 . Therefore, it is not surprising that we observed differences in valve size selection solely based on MDCT aortic annulus measurements by the observers who were not specialized in structural interventions. The same is true for reducing the risk of a rare but life-threatening aortic annulus rupture. The risk of annulus rupture is associated with high amounts of LVOT calcification and also with prosthetic valve oversizing 26 . All these aspects are not represented in the valve size selection recommendation sheet; they are solely based on clinical experience. There is a multitude of evidence that dedicated training can further improve operator performance [27][28][29] . Therefore respective programs should be defined and incorporated in relevant guidelines. Nevertheless the excellent intra-observer reproducibility of the aforementioned parameters for all observers is quite reassuring in that these clinical decisions were based on very robust data that can be accurately and safely assessed and then interpreted by an experienced physician. The software for structural heart disease used in the current study offers an excellent intra-and inter-observer reproducibility in regard to aortic valve measurements. This is likely due to its simplicity and the usability of the program as well as the good image quality of the MDCT technology, which, in contrast to echocardiography, is independent of the examiner. Taken together, successful pre-TAVR MDCT imaging should always be analysed by the actual implanting physician with subsequent valve size selection to allow for safe planning of the procedure based on individual anatomical and clinical patient data. In this regard, our study suggests that clinical experience is likely more important for adequate decision making as compared to the MDCT measurements which can be very accurately performed with generation of highly reproducible measurements after a brief introduction to current MDCT software programs.
Furthermore, borderline decisions on valve size selection will never fully rely on standardized measurement recommendation sheets but rather on the observer's experience. In clinical practice steps for appropriate selection of a device in borderline cases should always include: 1. a repeated analysis of the annulus area and annulus perimeter to rule out measurement errors; 2. the amount and distribution of calcification and the height of the coronary arteries should be taken into consideration; 3. in situations with large calcifications in the device landing zone ,the smaller valve should be chosen to reduce the risk of an annulus rupture; 4. if there is only a small amount of calcification, the larger valve size should be considered because of a resulting larger valve area and a lower risk of a paravalvular regurgitation; 5. if possible, in borderline situations a second experienced structural interventionalist should be involved with subsequent selection of the adequate valve size in a team approach.

Limitations
Some limitations of our study need to be addressed. First, our results are derived from a single vendor software package and may not apply to different analyses tools. However, the software package used is well established in clinical routine, and thus our results can be considered clinically meaningful. Second, the selection of the valve size was only based on the measurements of the annulus and the SOV and did not take the vascular access or coronary anatomy into account. However, since this was true for all observers, this bias can be considered a systematic bias. Third, each group of "differently experienced healthcare professionals" (aside from the structural interventionlists) were represented by one person only, all of whom had varying exposure to MDCT measurements before the study was started. Finally, our study included Medtronic CoreValve and the Edwards SAPIEN 3™ valves which were standard devices at the time of data collection.

Conclusions
MDCT planning and accurate valve size selection remains essential for TAVR without complications. Current state-of-the-art MDCT analysis software provides excellent reproducibility for anatomical measurements irrespective of the level of pre-existing experience. However, the highest levels of confidence in terms of valve size selection are achieved by implanting physicians, which can likely be attributed to observer experience, reflecting current clinical practice. Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.